{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Deep Learning model to detect suspiciously named files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "This notebook uses a pre-trained Deep Learning model to predict whether a processname is suspicious or not.\n",
    "\n",
    "Malwares and malicious programs such as ransomware often use tactics, techniques, and procedures (TTPs) such as copying malicious files to the local machine to propagate themselves across the network. A key indicator of compromise is that after a successful execution of the malware, it copies itself as an executable file with a randomly generated filename and places this file in one of the directories. Such techniques are seen in several malwares such as TrickBot. \n",
    "\n",
    "We develop machine learning model that uses a Recurrent Neural Network (RNN) to distinguish between malicious and benign processnames. The model is trained independently and is then made available for download. We use a character level RNN to classify malicious vs. benign processnames. Higher the `is_malicious_prob`, the more likely is the processname to be suspicious (between `[0,1]`). The threshold for flagging a processname as suspicious is set at `0.5`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 0 - import libraries\n",
    "At stage 0 we define all imports necessary to run our subsequent code depending on various libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "deletable": false,
    "name": "mltkc_import"
   },
   "outputs": [],
   "source": [
    "# This definition exposes all python module imports that should be available in all subsequent commands\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import string\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# global constants\n",
    "MODEL_DIRECTORY = \"/srv/app/model/data/detect_suspicious_processnames_using_pretrained_model_in_dsdl/detect_suspicious_processnames_using_pretrained_model_in_dsdl.pt\"\n",
    "\n",
    "# model parameters\n",
    "all_letters = string.ascii_lowercase\n",
    "n_letters = len(all_letters)\n",
    "n_hidden = 32\n",
    "n_categories = 2\n",
    "learning_rate = 5e-4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numpy version: 1.23.3\n",
      "pandas version: 1.5.0\n"
     ]
    }
   ],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "print(\"numpy version: \" + np.__version__)\n",
    "print(\"pandas version: \" + pd.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 1 - get a data sample from Splunk\n",
    "There are currently 2 ways to retrieve data from Splunk: Option 1 is to interactively pull data from Splunk into the DSDL Jupyter Lab environment. This is useful when the Splunk REST API is accessible from the Jupyter environment and a valid Splunk auth token is defined in the DSDL app. This option has advantages to quickly experiment with different Splunk SPL queries and further interactively work with the search results in Jupyter.\n",
    "\n",
    "### Option 1 - pull data from Splunk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dsdlsupport import SplunkSearch as SplunkSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07142129edfe4bb6b4159c9fbc7ec0d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(Textarea(value='| makeresults count=10 \\n| streamstats c as i \\n| eval s = i%3 \\…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "search = SplunkSearch.SplunkSearch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_time</th>\n",
       "      <th>feature_0</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>i</th>\n",
       "      <th>s</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.2428163280710578</td>\n",
       "      <td>0.2446760144084692</td>\n",
       "      <td>0.878678847104311</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.458672939799726</td>\n",
       "      <td>0.4403486642986536</td>\n",
       "      <td>0.5951908086426556</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.4268499561585486</td>\n",
       "      <td>0.70259142620489</td>\n",
       "      <td>0.6676120953634381</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.9911085437051952</td>\n",
       "      <td>0.7537638866342604</td>\n",
       "      <td>0.8423880040645599</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.2864162167534232</td>\n",
       "      <td>0.4240174447186291</td>\n",
       "      <td>0.7597004366107285</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.9403447071090341</td>\n",
       "      <td>0.10794003773480654</td>\n",
       "      <td>0.22778158355504274</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.6541405455209315</td>\n",
       "      <td>0.7657363642938435</td>\n",
       "      <td>0.07344387145712972</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.8196326880715787</td>\n",
       "      <td>0.34731235541403294</td>\n",
       "      <td>0.8556739939376712</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.3521619248203933</td>\n",
       "      <td>0.04771727975457907</td>\n",
       "      <td>0.07742329593747854</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2022-10-12T15:58:22.000+02:00</td>\n",
       "      <td>0.2476972616277635</td>\n",
       "      <td>0.5393188064917922</td>\n",
       "      <td>0.7553935945034027</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           _time           feature_0            feature_1  \\\n",
       "0  2022-10-12T15:58:22.000+02:00  0.2428163280710578   0.2446760144084692   \n",
       "1  2022-10-12T15:58:22.000+02:00   0.458672939799726   0.4403486642986536   \n",
       "2  2022-10-12T15:58:22.000+02:00  0.4268499561585486     0.70259142620489   \n",
       "3  2022-10-12T15:58:22.000+02:00  0.9911085437051952   0.7537638866342604   \n",
       "4  2022-10-12T15:58:22.000+02:00  0.2864162167534232   0.4240174447186291   \n",
       "5  2022-10-12T15:58:22.000+02:00  0.9403447071090341  0.10794003773480654   \n",
       "6  2022-10-12T15:58:22.000+02:00  0.6541405455209315   0.7657363642938435   \n",
       "7  2022-10-12T15:58:22.000+02:00  0.8196326880715787  0.34731235541403294   \n",
       "8  2022-10-12T15:58:22.000+02:00  0.3521619248203933  0.04771727975457907   \n",
       "9  2022-10-12T15:58:22.000+02:00  0.2476972616277635   0.5393188064917922   \n",
       "\n",
       "             feature_2   i  s  \n",
       "0    0.878678847104311   1  1  \n",
       "1   0.5951908086426556   2  2  \n",
       "2   0.6676120953634381   3  0  \n",
       "3   0.8423880040645599   4  1  \n",
       "4   0.7597004366107285   5  2  \n",
       "5  0.22778158355504274   6  0  \n",
       "6  0.07344387145712972   7  1  \n",
       "7   0.8556739939376712   8  2  \n",
       "8  0.07742329593747854   9  0  \n",
       "9   0.7553935945034027  10  1  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = search.as_df()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option 2 - push data from Splunk\n",
    "In Splunk run a search to pipe a dataset into your notebook environment. You utilize the `mode=stage` flag in the in the `| fit` command to do this. The search results are accessible then as csv file with the same model name that is defined in the `into app:<modelname>` part of the fit statement. Additionally, meta data is retrieved and accessible as json file. In the same way you can further work with the meta data object as it is exposed in the fit and apply function definitions below in stage 3 and 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| makeresults count=10<br>\n",
    "| streamstats c as i <br>\n",
    "| eval s = i%3 <br>\n",
    "| eval feature_{s}=0 <br>\n",
    "| foreach feature_* [eval \\<\\<FIELD\\>\\>=random()/pow(2,31)]<br>\n",
    "| fit MLTKContainer mode=stage algo=barebone_template _time feature_* i into app:barebone_template<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the into keyword (\"barebone_model\" in the example above) or set to \"default\" if no into keyword is present. This step is intended to work with a subset of your data to create your custom model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "deletable": false,
    "name": "mltkc_stage"
   },
   "outputs": [],
   "source": [
    "# this cell is not executed from MLTK and should only be used for staging data into the notebook environment\n",
    "def stage(name):\n",
    "    with open(\"data/\"+name+\".csv\", 'r') as f:\n",
    "        df = pd.read_csv(f)\n",
    "    with open(\"data/\"+name+\".json\", 'r') as f:\n",
    "        param = json.load(f)\n",
    "    return df, param"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "df, param = stage(\"barebone_template\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_time</th>\n",
       "      <th>i</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>feature_0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>1</td>\n",
       "      <td>0.749423</td>\n",
       "      <td>0.861980</td>\n",
       "      <td>0.063747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>2</td>\n",
       "      <td>0.328421</td>\n",
       "      <td>0.494055</td>\n",
       "      <td>0.578063</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>3</td>\n",
       "      <td>0.456588</td>\n",
       "      <td>0.135485</td>\n",
       "      <td>0.935327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>4</td>\n",
       "      <td>0.034384</td>\n",
       "      <td>0.457241</td>\n",
       "      <td>0.043726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>5</td>\n",
       "      <td>0.319659</td>\n",
       "      <td>0.867864</td>\n",
       "      <td>0.595515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>996</td>\n",
       "      <td>0.171754</td>\n",
       "      <td>0.162497</td>\n",
       "      <td>0.923288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>997</td>\n",
       "      <td>0.500971</td>\n",
       "      <td>0.899659</td>\n",
       "      <td>0.607057</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>998</td>\n",
       "      <td>0.209019</td>\n",
       "      <td>0.751987</td>\n",
       "      <td>0.469225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>999</td>\n",
       "      <td>0.163475</td>\n",
       "      <td>0.825084</td>\n",
       "      <td>0.158101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>1665565121</td>\n",
       "      <td>1000</td>\n",
       "      <td>0.208661</td>\n",
       "      <td>0.545487</td>\n",
       "      <td>0.872765</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          _time     i  feature_1  feature_2  feature_0\n",
       "0    1665565121     1   0.749423   0.861980   0.063747\n",
       "1    1665565121     2   0.328421   0.494055   0.578063\n",
       "2    1665565121     3   0.456588   0.135485   0.935327\n",
       "3    1665565121     4   0.034384   0.457241   0.043726\n",
       "4    1665565121     5   0.319659   0.867864   0.595515\n",
       "..          ...   ...        ...        ...        ...\n",
       "995  1665565121   996   0.171754   0.162497   0.923288\n",
       "996  1665565121   997   0.500971   0.899659   0.607057\n",
       "997  1665565121   998   0.209019   0.751987   0.469225\n",
       "998  1665565121   999   0.163475   0.825084   0.158101\n",
       "999  1665565121  1000   0.208661   0.545487   0.872765\n",
       "\n",
       "[1000 rows x 5 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'options': {'params': {'mode': 'stage', 'algo': 'barebone_template'},\n",
       "  'args': ['_time', 'feature_*', 'i'],\n",
       "  'feature_variables': ['_time', 'feature_*', 'i'],\n",
       "  'model_name': 'barebone_template',\n",
       "  'algo_name': 'MLTKContainer',\n",
       "  'mlspl_limits': {'handle_new_cat': 'default',\n",
       "   'max_distinct_cat_values': '100',\n",
       "   'max_distinct_cat_values_for_classifiers': '100',\n",
       "   'max_distinct_cat_values_for_scoring': '100',\n",
       "   'max_fit_time': '600',\n",
       "   'max_inputs': '100000',\n",
       "   'max_memory_usage_mb': '4000',\n",
       "   'max_model_size_mb': '30',\n",
       "   'max_score_time': '600',\n",
       "   'use_sampling': 'true'},\n",
       "  'kfold_cv': None},\n",
       " 'feature_variables': ['_time', 'feature_1', 'feature_2', 'feature_0', 'i']}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "param"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 2 - create and initialize a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "deletable": false,
    "name": "mltkc_init"
   },
   "outputs": [],
   "source": [
    "# initialize your model\n",
    "# available inputs: data and parameters\n",
    "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n",
    "\n",
    "class ProcessnameClassifier(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(ProcessnameClassifier, self).__init__()\n",
    "        self.hidden_size = hidden_size\n",
    "        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)\n",
    "        self.i2o = nn.Linear(input_size + hidden_size, output_size)\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "\n",
    "    def forward(self, input, hidden):\n",
    "        combined = torch.cat((input, hidden), 1)\n",
    "        hidden = self.i2h(combined)\n",
    "        output = self.i2o(combined)\n",
    "        output = self.softmax(output)\n",
    "        return output, hidden\n",
    "\n",
    "    def initHidden(self):\n",
    "        return torch.zeros(1, self.hidden_size)\n",
    "\n",
    "    \n",
    "def init(df, param):\n",
    "    model = ProcessnameClassifier(n_letters, n_hidden, n_categories)\n",
    "    model.load_state_dict(torch.load(MODEL_DIRECTORY, map_location=torch.device('cpu')))\n",
    "    model.eval()\n",
    "    \n",
    "    return model\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hyperparameter': 42.0}\n"
     ]
    }
   ],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "model = init(df,param)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 3 - fit the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "deletable": false,
    "name": "mltkc_fit"
   },
   "outputs": [],
   "source": [
    "# train your model\n",
    "# returns a fit info json object and may modify the model object\n",
    "def fit(model,df,param):\n",
    "    # model.fit()\n",
    "    info = {\"message\": \"model trained\"}\n",
    "    return info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'message': 'model trained'}\n"
     ]
    }
   ],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "print(fit(model,df,param))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 4 - apply the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "deletable": false,
    "name": "mltkc_apply"
   },
   "outputs": [],
   "source": [
    "# apply your model\n",
    "# returns the calculated results\n",
    "\n",
    "# Find letter index from all_letters\n",
    "def letterToIndex(letter):\n",
    "    return all_letters.find(letter)\n",
    "\n",
    "\n",
    "# Turn a line into a <line_length x 1 x n_letters>,\n",
    "# or an array of one-hot letter vectors\n",
    "def lineToTensor(line):\n",
    "    tensor = torch.zeros(len(line), 1, n_letters)\n",
    "    for li, letter in enumerate(line):\n",
    "        tensor[li][0][letterToIndex(letter)] = 1\n",
    "    return tensor\n",
    "\n",
    "\n",
    "def evaluate(model, line_tensor):\n",
    "    hidden = model.initHidden()\n",
    "\n",
    "    for i in range(line_tensor.size()[0]):\n",
    "        output, hidden = model(line_tensor[i], hidden)\n",
    "\n",
    "    return output\n",
    "\n",
    "\n",
    "def preprocess_txt(input_line):\n",
    "    input_line = input_line.split('/')[-1]\n",
    "    input_line = input_line.lower()\n",
    "    input_line = input_line.replace('.exe','')\n",
    "    input_line = ''.join(filter(str.islower, input_line))\n",
    "    \n",
    "    return input_line\n",
    "\n",
    "\n",
    "def apply(model, df, param):\n",
    "    is_malicious_prob_lst = []\n",
    "    for idx, row in df.iterrows():\n",
    "        input_line = row['text']\n",
    "        input_line = preprocess_txt(input_line)\n",
    "        \n",
    "        if len(input_line) == 0:\n",
    "            is_mal_prob = 1\n",
    "        else:\n",
    "            with torch.no_grad():\n",
    "                output = evaluate(model, lineToTensor(input_line))\n",
    "\n",
    "            output = torch.exp(output)\n",
    "            is_mal_prob = round(output[0][1].item(), 4)\n",
    "\n",
    "        is_malicious_prob_lst.append(is_mal_prob)\n",
    "    \n",
    "    output = pd.DataFrame()\n",
    "    output['is_malicious_prob'] = is_malicious_prob_lst\n",
    "\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     index\n",
      "0        0\n",
      "1        1\n",
      "2        2\n",
      "3        3\n",
      "4        4\n",
      "..     ...\n",
      "995    995\n",
      "996    996\n",
      "997    997\n",
      "998    998\n",
      "999    999\n",
      "\n",
      "[1000 rows x 1 columns]\n"
     ]
    }
   ],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "print(apply(model,df,param))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 5 - save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "deletable": false,
    "name": "mltkc_save"
   },
   "outputs": [],
   "source": [
    "# save model to name in expected convention \"<algo_name>_<model_name>\"\n",
    "def save(model,name):\n",
    "    with open(MODEL_DIRECTORY + name + \".json\", 'w') as file:\n",
    "        json.dump(model, file)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 6 - load the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "deletable": false,
    "name": "mltkc_load"
   },
   "outputs": [],
   "source": [
    "# load model from name in expected convention \"<algo_name>_<model_name>\"\n",
    "def load(name):\n",
    "    model = ProcessnameClassifier(n_letters, n_hidden, n_categories)\n",
    "    model.load_state_dict(torch.load(MODEL_DIRECTORY, map_location=torch.device('cpu')))\n",
    "    model.eval()\n",
    "    return model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 7 - provide a summary of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "deletable": false,
    "name": "mltkc_summary"
   },
   "outputs": [],
   "source": [
    "# return a model summary\n",
    "def summary(model=None):\n",
    "    returns = {\"version\": {\"numpy\": np.__version__, \"pandas\": pd.__version__} }\n",
    "    return returns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After implementing your fit, apply, save and load you can train your model:<br>\n",
    "| makeresults count=10<br>\n",
    "| streamstats c as i<br>\n",
    "| eval s = i%3<br>\n",
    "| eval feature_{s}=0<br>\n",
    "| foreach feature_* [eval &lt;&lt;FIELD&gt;&gt;=random()/pow(2,31)]<br>\n",
    "| fit MLTKContainer algo=barebone s from feature_* into app:barebone_model<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or apply your model:<br>\n",
    "| makeresults count=10<br>\n",
    "| streamstats c as i<br>\n",
    "| eval s = i%3<br>\n",
    "| eval feature_{s}=0<br>\n",
    "| foreach feature_* [eval &lt;&lt;FIELD&gt;&gt;=random()/pow(2,31)]<br>\n",
    "| apply barebone_model as the_meaning_of_life"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Send data back to Splunk HEC\n",
    "When you configured the Splunk HEC Settings in the DSDL app you can easily send back data to an index with [Splunk's HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector). Read more about data formats and options in the [documentation](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata).\n",
    "\n",
    "### Use cases\n",
    "- you want to offload longer running, possibly distributed computations that need to deliver results asynchroneously back into Splunk. \n",
    "- you might not want to present results back into the search pipeline after your `| fit` or `| apply` command. \n",
    "- you can easily utilize this approach for any logging purposes or other profiling tasks in your ML code so you can actively monitor and analyze your processes.\n",
    "\n",
    "### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dsdlsupport import SplunkHEC as SplunkHEC\n",
    "hec = SplunkHEC.SplunkHEC()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example to send 10 hello world events\n",
    "response = hec.send_hello_world(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEC endpoint http://host.docker.internal:8088/services/collector/event \n",
      "returned with status code 200 \n",
      "and response message: {\"text\":\"Success\",\"code\":0}\n"
     ]
    }
   ],
   "source": [
    "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example to send a JSON object, e.g. to log some data\n",
    "from datetime import datetime\n",
    "response = hec.send({'event': {'message': 'operation done', 'log_level': 'INFO' }, 'time': datetime.now().timestamp()})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEC endpoint http://host.docker.internal:8088/services/collector/event \n",
      "returned with status code 200 \n",
      "and response message: {\"text\":\"Success\",\"code\":0}\n"
     ]
    }
   ],
   "source": [
    "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Stages\n",
    "All subsequent cells are not tagged and can be used for further freeform code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
